1. Field of the Invention
A certain aspect of the present invention relates to an on-vehicle camera mounted on a vehicle and a technology for adjusting an image taken by the on-vehicle camera.
2. Description of the Related Art
An on-vehicle camera is mounted, for example, on the rear, front, or side of a vehicle and used to take images of scenes surrounding the vehicle and to display the images on a monitor provided in the vehicle. Such an on-vehicle camera allows a driver to see images of blind spots on a monitor and thereby assists the driver, for example, in driving a vehicle backward or in looking right and left at a blind intersection. Also, there is a known technology for superposing graphics, such as vehicle width lines indicating the width of a vehicle and distance reference lines indicating distances from the rear end of a vehicle, on an image taken by a camera and displaying the image together with the superposed graphics on a monitor. For example, this technology allows a driver to estimate a parking position or to intuitively understand a distance to an obstacle.
An image taken by an on-vehicle camera has to be adjusted according to the mounting position, height, and/or angle of the camera on a vehicle. For example, if the mounting position of an on-vehicle camera is off center in the width direction of a vehicle, it is necessary to correct an image taken by the on-vehicle camera such that the center line in the horizontal direction of the image on a monitor is aligned with a point on an extension of the center line in the width direction of the vehicle. Without the correction, the center in the width direction of the vehicle and the center in the horizontal direction of the image on the monitor are misaligned. This in turn makes it difficult for a driver to drive the vehicle. Similarly, to properly generate and display an overhead image as seen from above a vehicle by image processing, it is necessary to select proper image processing parameters according to the mounting height and/or angle of an on-vehicle camera. Also, when superposing graphics, such as vehicle width lines indicating the width of a vehicle and/or distance reference lines indicating distances from the rear end of the vehicle, on an image taken by an on-vehicle camera, it is necessary to adjust positions and angles of the superposed graphics according to the size of the vehicle and the mounting position of the on-vehicle camera.
In the related art, such adjustments are performed by the user. For example, objects are placed on a parking space to indicate the width of and distances from a vehicle, and with reference to the objects, the user adjusts the center position of an image, optimizes an overhead image, and adjusts the positions and angles of vehicle width lines and distance reference lines by using buttons or a touch panel display (see, for example, patent document 1). Adjustments that have to be done include horizontal position adjustment for correcting a shift of an image in the width direction of a vehicle, look-down angle adjustment for adjusting the look-down angle of an overhead image, width-line horizontal position adjustment for adjusting the center position of vehicle width lines, width-line distance adjustment for adjusting the distance between vehicle width lines, width-line angle adjustment for adjusting the angles of vehicle width lines, and distance reference line adjustment for adjusting the positions of distance reference lines. Obviously, it is troublesome for a user to make these complex adjustments.
Patent document 2 discloses a technology where display positions of graphics are adjusted based on the shape of a vehicle body and the mounting position of an on-vehicle camera that are input by a user. In the disclosed technology, display positions of distance marks indicating distances from the rear end of a vehicle are determined based on the mounting height of a camera. Generally, however, graphics to be superposed on an image include not only distance reference lines (corresponding to the distance marks in patent document 2) indicating distances but also vehicle width lines indicating the width of a vehicle. Therefore, it is not enough to determine only the display positions of distance marks.
In patent document 2, the angle (pitch angle) of a camera is fixed such that the bumper is in the sight of the camera and the mounting height (Y) is varied to determine the display positions of distance marks. However, an image taken by a camera may also be shifted (or deformed) due to the mounting position (X) of the camera in the vehicle width direction, the mounting angle (yaw angle) of the camera in the horizontal direction, and the rotation angle (roll angle) of the camera. Therefore, it is necessary to correct the image based on the amount of shift and also to adjust the display positions of graphics such as vehicle width lines and distance reference lines to be superposed on the corrected image. Also, if the mounting height of a camera is changed, a displayed overhead image may be distorted. Further, the distance between vehicle width lines to be superposed on an image may have to be varied depending on the width of a vehicle on which a camera is mounted.
Particularly, when a camera is to be retrofitted to a vehicle, i.e., when the type of vehicle the camera is to be mounted on and/or the mounting position of the camera are not predetermined, it is more important to correct images and adjust the positions of superposed graphics.    [Patent document 1] Japanese Patent No. 3551920    [Patent document 2] Japanese Patent No. 3448443
For better understanding of the present invention, a related-art on-vehicle camera is described below with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating a hardware configuration of a related-art on-vehicle camera. FIG. 2 is a drawing illustrating various mounting angles (pitch angles) of the on-vehicle camera.
As shown in FIG. 1, the related-art on-vehicle camera includes an imaging unit 10. The imaging unit 10 includes at least an imaging element and a lens. The imaging unit 10 obtains image data of a scene behind the vehicle by focusing light with the lens on the imaging element. The imaging element is, for example, implemented by a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The image data obtained by the imaging unit 10 are normally color (RGB) image data with a Bayer pattern, but may instead be monochrome image data.
The imaging unit 10 sends the obtained image data to an image processor 30. The image processor 30 includes an image processing unit 31 and a graphic superposing unit 32. The image processing unit 31 performs image processing on the image data sent from the imaging unit 10. The image processing, for example, includes Bayer interpolation, correction of chromatic aberration of magnification, YUV conversion, distortion correction, and overhead image generation. The graphic superposing unit 32 superposes graphics such as vehicle width lines and distance reference lines on the image data. More particularly, the image processing unit 31 and the graphic superposing unit 32 perform the above processing based on parameters (or adjustment values) stored in a user information storing unit 40.
The image processor 30 is, for example, implemented by an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and/or a digital signal processor (DSP). Alternatively, the image processor 30 may be implemented by a software program executed by a central processing unit (CPU).
An operations unit 20 is a user interface for correcting an image and may be implemented by buttons or a touch panel display. The operations unit 20 allows the user to adjust the horizontal position of an image and the look-down angle of an overhead image, and to set parameters of graphics such as vehicle width lines and distance reference lines. Image adjustment values and graphic adjustment values input by the user are stored in the user information storing unit 40. When the on-vehicle camera is turned on the next time, the adjustment values are read from the user information storing unit 40, and image data and graphics are adjusted based on the adjustment values. When the on-vehicle camera is turned on for the first time, factory default adjustment values are used.
The user information storing unit 40 is a rewritable nonvolatile memory and may be implemented, for example, by an electrically erasable programmable read-only memory (EEPROM) or a flash memory.
The processed image data with superposed graphics such as vehicle width lines and distance reference lines are sent to a digital-to-analog converter (DAC) 50. The DAC 50 converts the image data from digital to analog and outputs the converted image data as an NTSC (National Television System Committee) signal to an external display device 60 such as a monitor.
FIG. 3 is a drawing illustrating display modes of the on-vehicle camera that are switched by the operations unit 20. Here, it is assumed that the operations unit 20 includes two buttons A and B (not shown).
The on-vehicle camera mounted on the rear of the vehicle is used, for example, to assist driving in reverse. For example, the on-vehicle camera is turned on when the shift lever is shifted to the reverse gear (R) while the engine is running. When the on-vehicle camera is turned on, a normal view in a normal display mode shown in FIG. 3 is selected and an image of a scene behind the vehicle is displayed on the display device 60 (monitor) in the vehicle. In the normal display mode, pressing the button A switches views clockwise and pressing the button B switches views counterclockwise. When the button A is pressed while the normal view is selected, a top view is selected and an overhead image as seen from above the vehicle is displayed. When the button A is pressed again, a side view is selected and an image of scenes to the right and left of the vehicle is displayed. When the button A is pressed in the side view, a wide view is selected and an image with a view angle wider than that in the normal view is displayed. When the button A is pressed yet again, the normal view is selected again.
In the normal, wide, and top views, vehicle width lines indicating the width of the vehicle and distance reference lines indicating distances from the rear end of the vehicle are superposed on the image as shown in FIG. 4.
An adjustment mode of the on-vehicle camera is described below. Generally, an on-vehicle camera has an adjustment mode used before the factory shipment or when the on-vehicle camera is initially mounted on a vehicle to adjust images and superposed graphics based on the mounting position and angle of the on-vehicle camera and the type or width of the vehicle. For example, simultaneously pressing the buttons A and B for two or more seconds while in the normal display mode causes the on-vehicle camera to switch from the normal display mode to the adjustment mode. In the adjustment mode, for example, the button A is used to increase an adjustment value and the button B is used to decrease an adjustment value. Also in the adjustment mode, pressing the button A for two or more seconds switches adjustment menus clockwise and pressing the button B for two or more seconds switches adjustment menus counterclockwise. Simultaneously pressing the buttons A and B again for two or more seconds saves modified adjustment values in the user information storing unit 40 and causes the on-vehicle camera to return to the normal display mode.
When the on-vehicle camera is switched to the adjustment mode, an “image horizontal position adjustment” menu is selected. Pressing the button A while the “image horizontal position adjustment” menu is selected moves an image displayed on the monitor two pixels to the right. Meanwhile, pressing the button B moves the image two pixels to the left. Thus, it is possible to adjust the horizontal position of an image on the monitor by operating the buttons A and B. As described above, pressing either the button A or B for two or more seconds selects the next adjustment menu.
A “look-down angle adjustment” menu is used to change an image processing parameter corresponding to the vertical angle (pitch angle) of the on-vehicle camera with respect to the ground surface. For example, it is necessary to set different image processing parameters for vertical angles of 30, 45, and 60 degrees to properly generate and display a top view image (i.e., an overhead image). The image processing parameter is adjusted, for example, such that side lines defining a parking space in the overhead image become parallel to each other as seen from above the vehicle. If markers are placed in a parking space as described later, the image processing parameter is adjusted such that virtual lines connecting the markers form a rectangle. Here, the vertical angle of the on-vehicle camera with respect to the ground surface is calculated based on an assumption that the horizontal direction indicates 0 degrees (30 degrees and 45 degrees in FIG. 2 indicate exemplary vertical angles).
A “width-line horizontal position adjustment” menu is used to adjust horizontal positions of the vehicle width lines and the distance reference lines with respect to the image. A “width-line angle adjustment” menu is used to adjust the angles of the right and left vehicle width lines with respect to the image. A “width-line distance adjustment” menu is used to adjust the distance between the right and left vehicle width lines.
A “distance reference line adjustment” menu is used to adjust the positions of distance reference lines in the vertical direction on the monitor. In FIG. 3, three distance reference lines (0.5 m line, 1 m line, and 2 m line) are shown. The positions of distance reference lines may be adjusted independently or collectively.
Here, an exemplary method of adjusting an image is described. FIG. 5 is a top view of a parking space and a vehicle. In the example shown in FIG. 5, the mounting position of the on-vehicle camera is shifted from the center in the vehicle width direction. As a result, as shown in FIG. 6, the center line of a displayed image is not aligned with the center line of the vehicle. In this case, the user corrects the image using the operations unit 20 such that the center of the image in the horizontal direction is aligned with the center of the vehicle in the vehicle width direction. In some cases, the user may also adjust the rotation of the image and/or the look-down angle of the overhead image based on the mounting angle (yaw angle) in the vehicle width direction, the mounting angle (pitch angle) in the vertical direction, and the rotation angle (roll angle) about the optical axis of the on-vehicle camera. The user may further adjust graphics such as the vehicle width lines and the distance reference lines.
FIG. 7 is a top view of a parking space and a vehicle and is used to describe a case where more complex adjustments are performed on an image and graphics such as vehicle width lines and distance reference lines. As described with reference to FIG. 4, the vehicle width lines and the distance reference lines are superposed on image data obtained by the imaging unit 10. In a case where the on-vehicle camera includes a frame memory, adjustments as described below may be performed after taking a still image in a position as shown in FIG. 7 and moving the vehicle to a safer place.
In the related-art (see, for example, patent document 1), markers are placed in a parking space as shown in FIG. 7 and detailed adjustments of display parameters of an image and graphics are performed with reference to the markers. The markers are placed in positions at specific distances from the vehicle, in positions on the extensions of lines indicating the vehicle width, and in a position on the extension of the center line in the width direction of the vehicle. The user adjusts graphics by operating the operations unit 20 with reference to the markers. In the example shown in FIG. 7, the markers are placed in a position on the center line in the width direction of the vehicle, and at the intersections between the 50-cm, 1-m, and 2-m lines indicating distances from the rear end of the vehicle and the extensions of lines indicating the vehicle width. The user adjusts the graphics while looking at image data showing the markers. For example, the user adjusts, on the monitor screen, the positions of the vehicle width lines in the horizontal direction, the distance between the vehicle width lines, the angles of the vehicle width lines, and the positions of the distance reference lines. Although the 50-cm, 1-m, and 2-m lines are shown as distance reference lines in FIG. 7, the number and types of distance reference lines may vary. Two or more lines may be adjusted at the same time. For example, when adjusting the distance between the vehicle width lines, the positions of the right and left vehicle width lines may be adjusted at the same time. Alternatively, lines may be adjusted independently.
FIG. 8 is a table showing an example of initial parameters of vehicle width lines and distance reference lines. The table shows an example of factory default values indicating X-Y coordinates of initial and end points of the vehicle width lines and the distance reference lines as shown in FIG. 4. Values as shown in FIG. 8 are initially stored in the user information storing unit 40. The graphic superposing unit 32 reads the values from the user information storing unit 40 and based on the read values, draws and superposes the vehicle width lines and the distance reference lines on an image. The image and the superposed lines are displayed on the monitor as shown in FIG. 4.
The positional relationships between an image displayed on the monitor and graphics superposed on the image may change depending on the mounting position and angle of the on-vehicle camera. FIG. 9 shows an image on which graphics are superposed using the default values shown in FIG. 8. In FIG. 8, the vehicle width lines and the distance reference lines are not aligned with the markers placed on the ground surface in the parking space. In such a case, the user has to correct the misalignment by using the operations unit 20.
The user operates the operations unit 20 with reference to the markers placed in the parking space to adjust the horizontal positions and angles of the vehicle width lines and the distance between the vehicle width lines such that the vehicle width lines correctly indicate the vehicle width, and to adjust the positions of the distance reference lines such that the distance reference lines correctly indicate distances from the vehicle. More particularly, the user adjusts the vehicle width lines and the distance reference lines using the adjustment menus in the adjustment mode of the on-vehicle camera shown in FIG. 3.
With the “width-line horizontal position adjustment” menu, the user adjusts the horizontal positions of the vehicle width lines by simultaneously moving the right and left vehicle width lines in the horizontal direction on the monitor. With the “width-line angle adjustment” menu, the user adjusts the angles of the right and left vehicle width lines simultaneously such that the angles become symmetric on the monitor. With the “width-line distance adjustment” menu, the user moves the right and left vehicle width lines simultaneously to adjust the distance between the vehicle width lines such that they become symmetric on the monitor. With the “distance reference line adjustment” menu, the user adjusts the vertical positions of the distance reference lines individually or collectively. The graphic superposing unit 32 calculates intersections between the vehicle width lines and the distance reference lines such that they are correctly aligned.
FIG. 10 shows an image and superposed graphics displayed on the monitor after the above adjustments. FIG. 11 shows X-Y coordinates of the initial and end points of the vehicle width lines and the distance reference lines after the adjustments.
After making the adjustments, the user switches the on-vehicle camera from the adjustment mode to the normal display mode. As a result, the adjusted parameters are stored in the user information storing unit 40. When the on-vehicle camera is turned on next time, the image processing unit 31 and the graphic superposing unit 32 read the adjusted parameters from the user information storing unit 40 and display an image as shown in FIG. 10.
In the related-art, display parameters of an image and graphics such as vehicle width lines and distance reference lines displayed on the monitor are corrected or adjusted as described above. With the related-art configuration, however, the user has to perform complex and various adjustments by himself/herself.
According to a technology disclosed in patent document 2, a table associating vehicle types and mounting positions of a camera with display positions of distance marks is stored in a storage unit, the mounting height (Y) of the camera is determined based on vehicle information and a mounting position input via an operations unit, and the corresponding display positions of the distance marks are read from the storage unit based on the mounting height (Y). In patent document 2, however, it is assumed that the mounting angle (pitch angle) of the camera is fixed. Accordingly, patent document 2 is silent about how to deal with a case where the mounting angle (pitch angle) of a camera is varied. Also, patent document 2 does not disclose a method or a configuration for adjusting images and vehicle width lines. Thus, even with the technology disclosed in patent document 2, the user has to perform complex adjustments based on the mounting position, the mounting height, and/or the mounting angle of a camera.